chore: backfill email-domain member organization dates (CM-1107)#4064
chore: backfill email-domain member organization dates (CM-1107)#4064skwowet wants to merge 2 commits intoimprove/CM-1105from
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a one-off backfill script to infer and persist missing dateStart/dateEnd for existing email-domain memberOrganizations, using activity evidence and existing stint-inference logic, and queues affected members for affiliation recalculation.
Changes:
- Add DAL query helpers + types to find
email-domainmember orgs missing dates and derive per-org activity dates. - Add a backend script to run the backfill in batches and enqueue impacted members in Redis.
- Rename the Redis set key used by the script executor worker workflow that recalculates member affiliations.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| services/libs/data-access-layer/src/members/types.ts | Adds a typed result shape for email-domain org activity date rows. |
| services/libs/data-access-layer/src/members/organizations.ts | Adds queries for (1) memberIds with undated email-domain orgs and (2) org activity dates for stint inference. |
| services/apps/script_executor_worker/src/activities/block-project-organization-affiliations.ts | Renames the Redis set key used to queue/pop members for affiliation recalculation. |
| backend/src/bin/scripts/backfill-email-domain-member-organization-dates.ts | New one-off backfill script: batch processing, stint inference, DB updates/inserts, Redis enqueue. |
| backend/package.json | Adds an npm script entry to run the new backfill script. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
skwowet
force-pushed
the
improve/CM-1105
branch
2 times, most recently
from
4f9f391 to
68a79fc
Compare
Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export async function markMemberForAffiliationRecalc(memberIds: string[]): Promise<void> { | ||
| try { | ||
| await svc.redis.sAdd('queue:recalculate:members:affiliation', memberIds) | ||
| await svc.redis.sAdd('recalculate-member-affiliations', memberIds) |
There was a problem hiding this comment.
This changes the Redis set key used for the affiliation-recalc queue. If there are any members already enqueued under the previous key, they will no longer be processed after this deploy.
Consider supporting both keys temporarily (e.g., drain old -> new), or document/run a one-time migration to move existing set members so no queued work is lost.
Summary
Adds a one-off backfill script for existing
email-domainmember organization rows with missing dates.The script:
email-domainmember organizations wheredateStartanddateEndare null.activityRelations.memberId/platform/usernameto find activity-date evidence.Note
Medium Risk
Runs data backfills that mutate
memberOrganizationsrows and changes the Redis key used to enqueue/dequeue affiliation recalculation, which could impact downstream processing if any consumers still expect the old key.Overview
Adds a new backfill script (
backfill-email-domain-member-organization-dates.ts) and npm entry to findemail-domainmemberOrganizationswith nulldateStart/dateEnd, infer stint date ranges from verified email + activity evidence, and apply inserts/updates in a transaction before queuing members for affiliation recalculation.Extends the data-access layer with queries to (1) batch memberIds missing dates and (2) derive per-org activity dates via joins across
organizationIdentities,memberIdentities, andactivityRelations, plus a newEmailDomainMemberOrganizationActivityDatetype.Unifies the Redis set used for affiliation-recalc batching by switching the script-executor worker from
queue:recalculate:members:affiliationtorecalculate-member-affiliations.Reviewed by Cursor Bugbot for commit f834a6b. Bugbot is set up for automated code reviews on this repo. Configure here.